Theoretical Results on Reinforcement

We present new theoretical results on planning within the framework of temporally abstract reinforcement learning (Precup & Sut-ton, 1997; Sutton, 1995). Temporal abstraction is a key step in any decision making system that involves planning and prediction. In temporally abstract reinforcement learning, the agent is allowed to choose among "behaviors", whole courses of action that may be temporally extended, stochastic, and contingent on previous events. Examples of behaviors include closed-loop policies such as picking up an object, as well as primitive actions such as joint torques. Knowledge about the consequences of behaviors is represented by special structures called multi-time models. In this paper we focus on the theory of planning with multi-time models. We deene new Bellman equations that are satissed for sets of multi-time models. As a consequence, multi-time models can be used interchangeably with models of primitive actions in a variety of well-known planning methods including value iteration, policy improvement and policy iteration .